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NON-COMMUTATIVE LATTICE PROBLEMS 
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Abstract. We consider several subgroup-related algorithmic ques¬ 
tions in groups, modeled after the classic computational lattice prob¬ 
lems, and study their computational complexity. We find polynomial 
time solutions to problems like finding a subgroup element closest to 
a given group element, or finding a shortest non-trivial element of a 
subgroup in the case of nilpotent groups, and a large class of surface 
groups and Coxeter groups. We also provide polynomial time algorithm 
to compute geodesics in given generators of a subgroup of a free group. 
Keywords. Lattice problems, nilpotent groups, free groups, surface 
groups, Coxeter groups. 
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1. Introduction 

1.1. Motivation. In this paper, following [laiiaE] we continue our re¬ 
search on non-commutative discrete (combinatorial) optimization. Namely, 
we define lattice problems for an arbitrary algebraic structure and then study 
these problems together with their variations for an arbitrary group G. The 
purpose of this research is threefold. First, we approach lattice problems in a 
very different context by viewing them in the framework of classical algebra, 
thus facilitating a deeper understanding of the nature of these problems in 
general. Second, we try to unify and tackle several interesting algorithmic 
problems in group theory that are related to lattice problems. Third, we aim 
to establish a unified outlook on several group-theoretic problems within the 
framework of lattice problems. We refer to m for the initial motivation, 
the set-up of the problems, and initial facts on non-commutative discrete 
optimization. 

1.2. Non-commutative lattice problems. Let G be a fixed finitely gen¬ 
erated group with a fixed finite set of generators A. We fix the word metric 
dA on G relative to the generating set A and for s' e G by 1^1 we denote the 
length dA{g,^) of g in the word metric dA- A geodesic representing 5 is a 
shortest word in A U A~^ representing g in G. 

We always assume below, if not said otherwise, that elements of G are 
given by words in the generators A U A~ ^, and finitely generated subgroups 
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of G are given by their finite generating sets, where elements are represented 
by words in A U A~^. 

All algorithmic problems we consider in this paper are related to word 
metrics, geodesics, and distances between various subsets in G. The most 
fundamental one among them is the geodesic problem in G which requires 
for a given g € G to find a geodesic representing this g in G. This problem 
is well-studied in geometric group theory and there are many known results, 
but we mention only the following ones which clarify the nature of geodesics 
in groups that we consider here: geodesics in hyperbolic groups [5], Coxeter 
groups [IlSElIl], Artin groups [9], and various solvable groups |12j . 

Element-subgroup distance problem: Given a finitely generated sub¬ 
group H of G and an element g € G find an element h H which is closest 
to g in the word metric on G. 

One can immediately recognize that this is precisely the non-commutative 
version of the classical closest element problem in lattices. 

Subgroup distance problem: Given two finitely generated subgroups H 
and K of G find the distance between them, i.e., find two elements h £ H 
and k G K, not both trivial, with a shortest distance apart. 

Notice, that in the problem above it is very natural to consider not only 
subgroups, but cosets, or double cosets, etc. The most general and powerful 
formulation here would be about distances between rational subsets of G. 

Shortest element problem in a subgroup: Given a finitely generated 
subgroup H of G find a shortest nontrivial element in H in the word metric 
on G. 

Clearly, this is a general form of the classical shortest vector problem in 
lattices. 

Subgroup geodesic problem: Given a subgroup H of G generated by 
hi,... ,hn G G and an element g ^ G which belongs to H find the geodesic 
length of g with respect to the word metric on H relative to the generating 
set hi,..., hfi. 

Obviously, this is a very vast generalization of the initial geodesic problem 
in G and in general it is much harder then the initial one. 

1.3. Results and the structure of the paper. In Section we prove 
some preliminary results on complexity of algorithmic problems in nilpo- 
tent groups and discuss some results due to Schupp on Coxeter and surface 
groups. 

In Section we show that the closest element problem is decidable in 
polynomial time in free groups, virtually nilpotent groups, Coxeter and sur¬ 
face groups satisfying Reduction Hypothesis (all these groups are finitely 
generated). And in Section we show that for the same groups the shortest 
element problem is decidable in polynomial time as well. 
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In Section we prove that for rational subsets of a free group, given by 
deterministic automata, the Rational Subsets Distance Problem is decidable 
in polynomial time. In particular, this allows one to compute distances 
between subgroups or cosets in free groups. 

In Section we solve the subgroup geodesic problem in free groups in 
polynomial time. Notice, that if the word problem in G is decidable then 
the subgroup geodesic problem is also decidable (by brute force verification), 
furthermore, if the word problem in G is decidable in polynomial time and 
the subgroup is embedded isometrically then the brute force algorithm has 
an exponential running time. Thus the subgroup geodesic problem in surface 
groups, or limit groups, as well as in quasi-convex subgroups of hyperbolic 
groups, is decidable in exponential time. However, even in free groups solv¬ 
ing this problem in polynomial time is a highly non-trivial business. 

2. Preliminaries 

2.1. Nilpotent groups. In this section we prove some basic facts regard¬ 
ing algorithmic complexity of certain problems in nilpotent groups. These 
facts appear to be well-known, but we were unable to find original explicit 
estimates of complexity. For the sake of completeness we provide them here. 

Recall that in a group G so-called N-fold commutators on a set A <Z G are 
defined as follows. 1-fold commutator on A is any element of A. Inductively, 
A^-fold commutator is any element [u,v\, where u is an Tfold commutator, 
and V a j-fold commutator on A, where i j = N. 

Further, recall that a free class c rank r nilpotent group A^r,c with basis 
X = {xi,... ,Xr} possesses a so-called Malcev basis, that is a tuple of ele¬ 
ments Y = {jji,... ,ym) (m is bounded above by a polynomial in r of degree 
that linearly depends on c) such that 

(1) every i/i is a fej-fold commutator on X, with ki > kj whenever i > j', 
and 

(2) and every element g G Nr^c can be uniquely represented as 

9 = y?^---y^, a*GZ. 

The latter expression is called the Malcev normal form of g. The tuple 
[ai,... ,a 772 ] is called the Malcev exponent of g. We write [ui,..., U 772 ] — 
Maly( 5 f) (or just Mal( 5 ) when Y is fixed) and g = in what 

follows we assume for definiteness that pi = Xi for i = 1 ,..., r. 

The following lemma is well-known (see [SI Theorem 6.5]). 

Lemma 2.1. Let r,c be positive integers, and let yi,...,ym be a Malcev 
basis for the free nilpotent group Nr,c of class c and rank r. 

(a) Then there are polynomials pi,...,pm in variables 
..., 0 ^ 772 , • • •) [dm such that 
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(b) Further, if Q = ai = ... = ak or 0 = /?i = ... = j3k, then the 
eorresponding values of pi, i = 1,2 ... ,k + 1, are at + fti. 

(c) There are polynomials qi,... ,qm in variables n,ai,..., am sueh that 

ivT = yT n,aiez. 

Lemma 2.2. Let r,c he positive integers, and let yi,...,ym be a Malcev 
basis for the free nilpotent group of class c and rank r. There is an 
algorithm that given a word g in free generators of computes the Malcev 
normal form of g in a time polynomial in the word length | 5 r| of g. 

Proof. We prove the following statement by induction in k: if re is a group 
word in variables yk, yk+i, ■ ■ ■, ym, 1 < k < m, then its Malcev normal form 
yf^ ■ ■ ■ yrtP t)e computed as a group word in a time polynomial in the 
word length of w. 

Base of induction k = m is obvious. Suppose the statement holds for 
k = j P 1, and prove it for k = j. Given a word w in variables yj,... ,ym, 
represent it as 

/3i 1^2 

W = wiy^^W2yj • • • wiy^^Wi^i, 

where rci,..., are (possibly trivial) group words in variables 

yj+i,...,ym and /3i,...,/3£ are nonzero integers. Note that both i and 
\wi\ are bounded by |r(;|, so after a polynomial time computation we can 
assume that all Wi are given by their Malcev normal forms. Then we “push” 
all occurrences of yj to the left, starting with the rightmost occurrence, as 
follows. 

• At the first step, we apply Lemma [2.IK a) to compute Malcev normal 
form of W£-yj‘. Note that by Lemma|2.1[b), this Malcev normal form 

is a word y^^w'^, where is a group word in variables yj+i,..., ym- 
This allows us to represent w as 

W = Wiy^ • • • Wi-iyj 

Note that the length of w'^ is polynomial in |t(;| by Lemma 

• Subsequently, having obtained 

w = wiy^^ ■ ■ ■ • • • w'^wi+i, I < i < £ - I, 

we apply Lemma |2.lK a,b) to compute Malcev normal form of wg-i ■ 
yhe-t+---+he ^ obtaining 

w = wiy^d ... ■ ■ ■ w'gwg+i, 

Note that the length of is polynomial in |r(;| by Lemma 

• Repeating the above step £ — 1 < |r(;| times, we arrive at 

/31 02 • “1“ 01 f f f 

= Uj W 1 W 2 • • • w^W£j^i. 

Now it is only left to observe that the word w[w 2 ■ ■ ■ w'^wg^i in vari¬ 
ables yj+i ,..., ym is of polynomial length in |u;|, so by the induction 


2.1 


2.1 
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assumption its Malcev normal form can be computed in polynomial 
time. 

Note that the degree of polynomial that bounds time complexity of the 
above procedure (possibly) grows as j decreases, but since m only depends 
on r, c, the degree of polynomial that bounds time complexity of the resulting 
algorithm ultimately depends only on r, c. □ 


Remark 2.3. 


( 1 ) 

( 2 ) 

(3) Lemmas 2.1 


The above procedure can be significantly optimized by taking into 
account structure of the Malcev basis Y (see HU)- 
It immediately follows that the exponents in Mal(g) are bounded by 
a polynomial in (that depends on r, c). 

2.2l 2.4 with minimal changes are also true for ap¬ 


propriate bases of arbitrary finitely generated nilpotent groups by 
essentially the same argument (see HD)- 


Lemma 2.4. Let r,c be positive integers, and let yi,... ,ym be a Malcev 
basis for the free nilpotent group of class c and rank r. There is an 
algorithm that, given elements gi,... ,gn G Nr^c ond exponents ki,... ,kn G 
Z, computes the Malcev normal form of the element g^^ ■ ■ ■ g^ in a time 
polynomial in N = n + Yl \9j\ + Yl \^j\- 


Proof. Note that the word length of an element g9 


’j\ < 


is bounded by \gj 

N'^ . Further, since n < N, the word length of g^^ ■ ■ ■ g^" is bounded by 
■ N = N'^. By Lemma 2.2, the Malcev normal form of such element can 
be computed in a time polynomial in N^. □ 


The “noncommutative Gauss” algorithm for solving membership problem 
in nilpotent (or, more generally, polycyclic) groups is well-known [TO]. In 
the following lemma we investigate the complexity of this algorithm. 


Lemma 2.5. Subgroup membership problem in a finitely generated nilpotent 
group is decidable in polynomial time. 


Proof. Since every subgroup of a finitely generated free nilpotent group is 
finitely generated, it is enough to prove the statement in the case of a free 
nilpotent group. 

Let Nrc be the free nilpotent group of rank r and class c. Since the 


Malcev normal form can be computed in polynomial time by Lemma 2.2 


we assume that a subgroup elements h,hi,... ,hn are given by their Malcev 
normal forms. For a tuple hi,..., h^, we form the coordinate matrix A, that 
is an n X m matrix whose f-th row is the Malcev exponent of hp. 


( 


A = 


ail 0^12 • • • O'lr. 


y Qlnl ^n2 


CXn 
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where hi = j We say that the matrix A is in 

triangular form if it has the following properties (vTi denotes so-called pivot, 
i.e., the position of the first nonzero element in row i): 

(i) All rows of A are non-zero (i.e. no hi is trivial). 

(ii) TTi < 712 < ... < TTs (where s is the number of pivots). 

The tuple hi,... ,hn is called full if the corresponding matrix is triangular 
and in addition 


(iii) H n (oj, Oj+i,..., Om) is generated by {hj \ -Kj > i}, for all 1 < i < m. 

In (pll|), note that {hj \ Hj > 1} consists of the elements that have 0 in 
their first 7 — 1 coordinates. It follows from Lemma |2.1[ b) that (iii) holds 
for a given i if and only if the following property holds. 




and h^’‘ 


(iii)’ For all 1 < A: < j < s with tt^ < i, 

{hi\l>k). 

To solve membership problem for a given input {h, hi 
by forming the coordinate matrix Aq for the tuple hi,... 


are elements of 


..., hn) we start 
hn- We produce 

matrices Ai,..., Ag, with s the number of pivots in the triangular full form 
of Aq, such that for every k = 1,..., s the first columns of A^ form a 
matrix satisfying ([^, and the condition (iii) is satisfied for all i < vr^+i, so 
that Ag, upon discarding trivial rows, is the triangular full form of Aq. Here 
we formally denote vr^+i = m -|- 1. 

Let Ak-i, A: > 1, be constructed. Below we construct the matrix A^, 
starting by setting Ak = A^-i- Below we let /ii,/i 2 ,... denote the group 
elements represented by the corresponding rows of the matrix Ak, and aij 
the entry (i,j) of Ak- 

First, we identify the column of the next pivot vr = vr^, that is the first 
column with at least one nonzero entry in rows i > k. Compute a linear 
expression of d = gcd(Q;fc^,..., amr)- 


d — IkC^kiT T ■ ■ ■ T InO^r, 


Using Lemma 2.4 we compute the Malcev exponent of the group element 

hn+l = hjf ■ ■ ■ , 

Mal(/lfi-|_i) — [0, . . . , 0, d — CXn-lljTT+lj • • • ) CIn+l,m]' 

Then we 

(1) add the above row to the matrix Ak', 

(2) for each i = k,... ,n, replace A-th row by Mai (jii ■ ; 

(3) rearrange rows k,... ,n + 1 of the obtained matrix so that the only 
nonzero entry in the first column in those rows is in the row k. 

Note that operations (1)~(3) above preserve preserve the group generated 
by rows of the matrix, and by Lemma 2.1 a,c) can be done in polynomial 


time in terms of the entries of Ak-i, and that the property (|^ with i < TTk 
holds for Ai by Lemma |2.lKb). 
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To obtain (iii) for i < vrfc+i, we identify the next pivot Tik+i, setting 
TTfc+i = m + 1 if TTfc is the last pivot. We now ensure condition (iii) for 
i < TTk+i. Observe that operations (l)-(3) above preserve {hj | vrj > i) for 
all i < TTk- Hence (iii) holds in for i < iTk since it holds in A^-i for 
the same range. Now consider i in the range It suffices to 


provide (iii)’ for all j > k. 

(p^’, we notice that G {h^ \ t > k) ii and 


G {he, \ (. > k). Further, note that the subgroup generated 


To obtain 
only if [hj, 
by the set 

Sj — {1, /ij, [hj , /ifc],..., [hj , hk )...) hk\ }, 
where hk appears m — Hk times in the last commutator, is closed under 
commutation with hk since if hk appears more than m — iTk times then 
the commutator is trivial. An inductive argument shows that the subgroup 

{Sj) coincides with (/i T | 0 < ^ < m — vr/j). Similar observations can be 


made for conjugation by Therefore, appending rows Coord(/iJ'') for all 
1 < [i[ < m — TTk and all A: < j < re + 3 delivers (hi)’ for all j > k. Note 
that (hi)’ remains true for i < TTk- 

s < m. Discarding trivial rows, 


hi. 


The process terminates at a matrix A, 
we obtain a matrix A that satisfies ([^-(iii), whose rows generate the same 
subgroup as those of Aq. Finally, observe that for a triangular full matrix 
A, checking whether h & H can be done straightforwardly. Indeed, let 
Mal(/i) = (ai, ... , am ) and let be the hrst nonzero coordinate of h. 
Then, by Lemma 2.1 and property (iii) of A, /i G H if and only if h^^^ = 
hi^aTr/ain ^ particular, if ciTr is not a multiple of aijr, then h ^ H). 

Proceed “left to right”, successively eliminating entries of h^'^\ If at any step 
the elimination fails, h ^ H. If the process terminates at a trivial h^^\ then 
h £ H, and moreover, a representation h = ■ ■ ■ h^f has been found. □ 


Remark 2.6. Observe that the above algorithm is presentation-uniform, 
that is, for a fixed r and c, given a presentation with at most r generators of 
a class at most c nilpotent group, and input of the membership problem, it in 
polynomial time decides membership in the group given by the presentation 
for the given elelements. 


2.2. Surface groups and Coxeter groups. In |16) . Schupp gives an ana¬ 
logue of Stallings graphs for free groups in the case of Coxeter and sur¬ 
face groups satisfying certain conditions. Remarkably, the corresponding 
algorithms have polynomial time complexity. To formulate the respective 
statements, we need to provide certain small cancellation conditions. 

Recall that a Coxeter group G is a group with presentation 

(1) G = (A I R) = (ai,...,an I a-, {aiaj)"^^\i^j), 

where rriij = rriji > 1 and we may have rriij = oo (which denotes the 
absence of a defining relator between ai and Oj). The above presentation is 
referred to as the standard presentation of the Coxeter group G. For each 
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1 < z < n, let Pi denote the number of indices j ^ i s.t. rriij < oo. Also set 

Pij = max{pj,/9j}. 

We say that a Coxeter group G given by its standard presentation satisfies 
the Reduction Hypothesis if in n > 3 and each niij > 4, and there is a 
subset CCA such that every defining relator contains a generator 

from C and satisfies the following condition: 

(1) If both Qi, Oj G C, then mij > \pij] 

(2) If Oj G C and Oj ^ C then mij > 2pi. 

We say that a surface group G given by its standard presentation satisfies 
the Reduction Hypothesis if the standard defining relator has length at least 
8, i.e. the genus is at least 2 in the orientable case and is at least 4 in the 
non-orientable case. 

Theorem 2.7 (Schupp, |16]1. There is a fixed quadratic time algorithm 
which, when given the standard presentation of a Coxeter group G = {A \ R) 
satisfying the Reduction Hypothesis, a tuple {hi ,..., hm) of generators for a 
subgroup H, calculates the graph A{H) which, in particular, is the graph of 
a finite state automaton which accepts a Dehn reduced word u if and only if 
uGH. 

Theorem 2.8 (Schupp, |16]). There is a fixed quartic time algorithm which, 
when given the standard presentation of a surface group of genus at least 

2 in the orientable case and at least 4 in the non-orientable case, and a 
tuple {hi,..., hm) of generators for a subgroup H, calculates the graph A(i7) 
which, in particular, is the graph of a finite state automaton which accepts 
a Dehn reduced word u if and only if u G H. 

3. Closest Element Problem 

Theorem 3.1. The closest element problem in free groups of finite rank is 
solvable in polynomial time. 

Proof. Let hi,h 2 ,... ,hn G F he words in free generators of a free group F. 
One can construct in polynomial time the Stallings graph A of the subgroup 
H = {hi,..., hn) of F. By 1 a we denote the base vertex of the graph A. 
Let r be a linear graph labeled by a freely reduced word representing g, 
with initial vertex Ip and terminal vertex tr. Attach L to A by identifying 
Ir and 1 a and perform Stallings foldings, marking the vertices of L that 
get identified with Ia- Let A^ be the obtained graph. Let v be the farthest 
marked vertex along L. We claim that the group element h„ G H read as a 
label of the path in L from Ip to v is the closest to g element in H. Indeed, 
let h G H be such that g = hg' . Then the element g' is readable in A^ as 
a label of a path from 1 a to tr- By construction, g' cannot be shorter than 
the length of the path from u to tr in L. □ 

Note that it is important for the above argument that the free group 
is given by its free generators. It is similarly important that in the below 
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theorem the Coxeter or the surface group involved in the statement is given 
by its standard presentation. The case of an arbitrary presentation remains 
open. 


Theorem 3.2. There is polynomial time algorithm that, given a standard 
presentation of a Coxeter group or a surface group G satisfying the Reduction 
Hypothesis, and elements g,hi,... ,hm £ G, finds the element h & H = 
{hi,..., hm) closest to g. 


Proof. Similarly to the free group case, we consider the graph that consists of 
a bouquet of loops reading hi,, hm, and the acyclic graph T that accepts 
all Dehn reduced forms of g. We apply procedure described in m in the 
proof of Theorems 2.7 and 2.8 to obtain the folded version of this graph, A. 
Now we find the shortest reduced word labeling a path from a vertex of T 
identified with 1 a to the terminus of image of T in A. Since geodesic words 
are Dehn reduced, this word will represent the closest to g element of H. □ 


Corollary 3.3. Let G be a Coxeter group or a surface group given by its 
standard presentation satisfying the Reduction Hypothesis. Then the Closest 
Element Problem in G is in P. 


Proof. Follows immediately from Theorem 3.2 


□ 


Theorem 3.4. Let G be a a finitely generated virtually nilpotent group. 
Then the Closest Element Problem in G is polynomial time decidable for an 
arbitrary subgroup of G. 


Proof. Indeed, let 77 be a given subgroup of G and g £ G. By a theorem 
of Wolf [18] the growth of G is polynomial, i.e. the ball TItv of radius N in 
Cayley graph of G centered at 1 contains polynomially many (in terms of 
N) elements of the group G. Moreover, by [131 Proposition 3.1] there is a 
P-time algorithm to list all group elements contained in Bat. Now, note that 
the distance from g to H cannot exceed ji^j. Set A = j^rj and find the closest 
to g element of H by bruteforcing elements of the form gb, b £ Bj^. Since the 
size of Bat is polynomial and the membership problem in finitely generated 


virtually nilpotent groups is decidable in polynomial time by Lemma 2.5 
this can be done in polynomial time. □ 


4. Distance between rational subsets 

In this section we consider the following problem. 

Rational subsets distance problem: Given two rational subsets R and 
S of G, find the distance between R and S, i.e., find two elements r £ R 
and s £ s with a shortest distance apart. 

Theorem 4.1. Subgroup distance problem in a free group is decidable in 
polynomial time. 






NON-COMMUTATIVE LATTICE PROBLEMS 


10 


Proof. Given subgroups H, G of the free group F, we construct in sub¬ 
quadratic time m their folded Stallings’ graphs ri,r 2 , respectively. Then 
we build the product graph A = Ti x r 2 . Consider the connected com¬ 
ponent Aq of the base vertex 1 a = (Ipi, lr 2 )- We may assume that Aq is 
a tree (otherwise F[ D G is nontrivial). For each vertex v of Aq, let be 
the element of the free group read along the reduced path from 1 a to v. 
Further, let and be the elements that realize distance from fy to H, G, 
respectively, that is, fyhy G H, fygy G G and \hy\, minimal. Note that 
for each v elements hy , gy can be found in polynomial time by Theorem |5.1[ 
Bruteforcing all vertices v of Aq, find v = vq with with minimal \hy \ + \gy\. 
We claim that the shortest distance between Ff an G is attained on the pair 
of elements /yghy^, fv^dvo- 

Indeed, let 5 G G and h G H he the pair with minimal \g~^h\. Then 
h = wh', g = wg', where there is no free cancellation in the product {g')~^h'. 
Then the element w is readable as a label of a path in A from 1 a to some 
vertex v. Since \g~^h\ = \g'\ + \h'\, the claim follows. □ 

Note that the key feature of Stallings graphs in the above argument is that 
they are deterministic automata. If a non-deterministic automata describ¬ 
ing rational subsets of a free group are given as an input, one can of course 
produce deterministic ones and apply a similar argument. However, pro¬ 
ducing a deterministic automaton out of a non-deterministic one may result 
in an exponential blow-up in size, which invalidates polynomial complexity 
estimate. While we are not aware of a polynomial algorithm in general case, 
we can, of course, accept deterministic automata as an input and solve the 
corresponding problem in polynomial time. 

Theorem 4.2. For rational subsets given hy deterministic automata, the 
Rational Subsets Distance Problem in a free group is decidable in polynomial 
time. 


Proof. The proof repeats that of the subgroup case (Theorem 4.1) with 
minor adjustments. □ 


As an immediate corollary we, for example, obtain that the shortest dis¬ 
tance between cosets in a free group is solvable in polynomial time. 


5. Shortest element problem 

Theorem 5.1. The shortest element problem in free groups of finite rank 
is in P. 

Proof. For a subgroup FF of a free group given by elements represented by 
words hi,... ,hn, we construct its Stallings graph F with initial vertex Ir- 
Then finding the shortest nontrivial element in H is equivalent to finding 
shortest loop at Ir, which is well known to be polynomial time. □ 

Theorem 5.2. There is polynomial time algorithm that, given a standard 
presentation of a Coxeter group or a surface group G satisfying the Reduction 
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Hypothesis, and elements hi,, hm £ G, finds the shortest element h G 
H = (III, . . . , hm) ■ 


Proof. Similarly to the free group case, we consider the graph A{H) provided 
by Theorem |2.7[ and find the shortest loop with a reduced label at Ir- □ 


Corollary 5.3. Let G be a Coxeter group or a surface group given by its 
standard presentation satisfying the Reduction Hypothesis. Then the Short¬ 
est Element Problem in G is in P. 


Proof. Follows immediately from Theorem 5.2 


□ 


Theorem 5.4. The shortest element problem in a finitely generated virtually 
nilpotent group G is in P. 


Proof. The proof is based on the same observation as that of Theorem 3.4 
Indeed, let FT be a subgroup of G given by words /ii,..., in generators of 
G. Let N denote the total length of these words. By a theorem of Wolf [18j 
the growth of G is polynomial, i.e. the ball of radius N in Cayley 
graph of G contains polynomially many elements of group G. Moreover, 
by da Proposition 3.1] there is a P-time algorithm to list all group elements 
contained in B]\f. Note that H contains nontrivial elements if and only if at 
least one of the words hi,... ,hn is nontrivial, i.e. if and only if Bjv contains 
at least one nontrivial element of H. Now, since the membership problem in 
finitely generated virtually nilpotent groups is decidable in polynomial time 
by Lemma [2.5[ we can find the shortest element in G in polynomial time by 
brute search. □ 


Remark 5.5. The argument above is based on the fact that finitely gen¬ 
erated virtually nilpotent groups have polynomial growth. By Gromov’s 
theorem [7] the converse is also true, i.e., polynomial growth implies virtual 
nilpotence, so the argument does not apply to any wider classes of groups. 


6. Subgroup geodesic problem in a free group 


In this section we discuss a subgroup geodesic problem for a finitely gen¬ 
erated free group F = F(X). It can be formulated as follows. Given words 
hi,..., hm, h € F express h as an element in H = {hi,..., hm) in an optimal 
way, i.e., exprees h as a product: 


with the least number of factors k. We prove that the problem can be solved 
in polynomial time. Our main tool is an Ff-digraph with an additional 
labeling function. Formally, we work with a tuple {V,E,p.,iy), where: 

• {V, E) defines a directed graph. 

• pi-. E^ X^. 

• u -. E ^ F{hi,..., hm), where hfs are formal letters that stand for 
the generators of H. 
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To achieve polynomial time complexity we represent the values of v using 
straight line programs (see [IS]). For an edge e G E with o(e) = ui, t{v 2 ) = 
V2 fJ-ie) = X and p(x) = u we often use the following notation: 


-r* Ti 



The edge V 2 ^ ^ vi is called the inverse of e = ^ U 2 and is denoted 

by e~^. A path p in a graph T is a sequence of consecutive edges ei,..., e^; 
it has labels ^(p) = /i(ei)... ^(e^) and v{p) = p(ei) ... p(efc). A circuit in 
T at u G F is a path p such that o{p) = t{p) = v. 

The initial step in the algorithm is to construct the graph Tq = 
Bouquet(hi,..., hm)- For every word hi = yi.. .yn G F (where yj G 
define a cyclic graph Th^ = {V, E), where: 

y = {0 ,... ,n - 1} 

and 

E ={i - i \ 1 < i < n}U {n - 1 0} 

U{z i — 1 I 1 < z < n} U {0 n — 1} 

U{i ^ z I 0 < i < n}. 

The graph Bouquet(hi,hm) is obtained by merging the graphs 
... ,Th^ at the vertex 0. The vertex 0 is designated as the root of 
bo. 



Figure 1. Bouquet(xi,xiX 2 ). Inverse edges omitted. 

Let 7 : T(hi,..., hm) —)• hh be an epimorphism given by 7 (hj) = hj. The 
next lemma follows from the definition of Fq. 

Lemma 6.1. Let p be a circuit at 0 in Fq. Then fj,(p) = 7 (z/(p)). □ 

We say that a path p in F is reduced if p does not involve a segment ee“^. 
Reduction of a path in F is a process of removing all segments ee“^. It is 
not difficult to see that the result of path-reduction is unique and does not 
depend on a particular choices of removals. 

We say that a pair of consecutive edges ei = ui —)> V 2 and 62 = U 2 —> V 3 
in F is a potential bypass if: 
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• |/i(ei)|u(e 2 )| < 1 and; 

• r does not contain an edge e = fi —)> ^3 such that ^(e) = fi{ei)fi{e 2 )■ 
Algorithm described below modifies the initial graph Fq and produces a 
sequence of graphs: 

Fo c Fi c ... c r„ 

where Fj+i is obtained from Fj by adding a single edge e* for some potential 
bypass 61,62 at Step[^ We point out that the output of Algorithm may 
depend on choices it makes at Step 


Algorithm 1 Completion: F' = Complete(F) 

Require: A bouquet graph F = {V, E) as above. 

Ensure: A graph F' = (C, E') with E C E'. 

1: while F contains potential bypasses do 

2 : Find a potential bypass 61,62 with the least \v{ei)u{e 2 )\- 

3: Add a new edge e = o(ei) —)• t{e 2 ) 

4: Put fi(e) = fj,(ei)fi(e 2 ). 

5: Put p( 6 ) = Z.^( 6 i)p( 62 ). 

6: end while 
7: return F. 



Figure 2. Complete graph for {xi,xiX 2 }. Inverse edges omitted. 


Lemma 6.2. Let F' = Complete(F). Let p be a circuit at the origin in F'. 
Then fi{p) = 7 (u(p)). 


Proof. We prove the statement by induction for every graph Fj. By Lemma 


6.1, p{p) = j{n{p)) holds for every circuit in Fq. Assume it holds for every 
and consider any circuit p in Fj. Let 6j be the edge in Fj 
If p does not involve 6j then we may think that p belongs 


circuit in Fj_i 
added to Fj_i. 

to Fj_i and hence p{p) = ^{n{p)). If p involves the edge Cj then replacing 
every occurrence of 6j by a subpath e'e'' we obtain a circuit p' satisfying 
p{p) = p{p') and v{p) = ^{p') and which does not involve 6j. Therefore, 
p{p) = p{p') = 7(u(p')) = n 
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We say that a path p in To defines a shortcut if 

(51) |p(p)| < 1; 

(52) u{p) is reduced; 

(53) |p(p)| is the least possible among the paths with the same endpoints 
and the label p(p). 

Clearly, ifp defines a shortcut, then any its subpath q with \ fj,{q)\ < 1 defines 
shortcut. We say that an edge e in F = Complete(ro) is a shortcut for a 
path p in To if p defines a shortcut and o(e) = o(p), t{e) = t{p), p(e) = p(p), 
and v{e) = p(p). 

The below lemma asserts that T contains all shortcuts for paths in Fq 
and, moreover, each edge in F is a shortcut for some path in Fq. 

Lemma 6.3. Let hi,...,hm £ \ Tq = Bouquet (/ii,/im); and 

F = Complete(Fo) . The following holds. 

(El) For every edge e in T there exists a pathp in Fq such that o{p) = o(e), 
t{p) = t{e), p(p) = p(e), and v{p) = v{e). 

(E2) Furthermore, every edge e = ui —>■ n 2 G F is a shortcut for some 
path p = Pe in Fq. 

(E3) If a path p from vi to V 2 in Tq defines a shortcut, then F contains 
an edge e such that p{e) = fi{p) and \i'{e)\ = |z^(p)|. 

Proof. Clearly (El) holds for every edge in Fq. Assume that (El) holds for 
every edge in Fj_i and consider the edge e G Fj — Fj_i, i.e., the potential 
bypass for edges e',e" added on the ith step to Fi_i. By the inductive 
assumption there exist paths fi,p” for e',e" satisfying (El). Then the path 
p'p" is a required path for e. Thus, (El) holds. 

It is straightforward to check that for every edge e in Fq the path pe = e 
(consisting of a single edge e) is the required path. Hence, (E2) holds for 
Fq. Assume that (E2) holds for every edge in Fi_i and consider the edge 
e G Fj — Fj_i, i.e., the potential bypass for edges e',e" added on the ith 
step to Fj_i. By the inductive assumption there exist paths p' ,p" for e' ,e'' 
satisfying (E2). If the path p = p'p” does not define a shortcut, then there is 
a shorter (reduced) path g in Fq with the origin o(e), the terminus t(p), and 
the label pi{q) = p{e) defining a shortcut. Since \p{q)\ < 1, we can split q into 
two nontrivial segments q = qiq 2 each defining a shortcut. Since Algorithm 
[^on step chooses a potential bypass with the least z^-value, it follows that 
Fj_i does not contain shortcuts for a path qi or q 2 . Assume without loss 
of generality that Fi_i does not contain a shortcut for qi. Proceeding as 
above for q, we split the path qi into two nontrivial segments qi = qiiqu 
each defining a shortcut. Since instead of adding a shortcut for qi Algorithm 

adds the edge e* it follows that Fj_i does not contain a shortcut for gn 
or a shortcut for qi 2 , say qu. And so on. Proceeding in the same fashion, 
we eventually get a path q' in Fq of length 1 (i.e., q' is an edge) for which 
there is no shortcut. But that is impossible because edge Fq is a shortcut 
for itself. The obtained contradiction proves that e satisfies (E2). 

One can prove (E3) the same way as (E2). □ 
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Algorithm 2 Geodesic: u = Geodesic(/ii,..., hk-, w). 

Require: {hi, ..., hm} C Fr\ {e} and w = G F. 

Ensure: A shortest factorization of w in hj’s. 

1: To = Bouquet(hi,..., hm)- 
2 : r = Complete(Fo). 

3: 5 = {(0,e)}cy 
4: for j = 1 to n do 
5: Put S' = 0. 

6 : for all {v, n) G 5* do 

7: for all e = n —)> n' in r s.t. ^(e) = do 

8: Put S" = 5'U {(n', ttp(e))}. 

9: end for 

10: end for 

11: Put 5 = 0. 

12: for all X G P do 

13: if there exists {v, u) G S' then 

14: Add a pair {v, u) from S' with the shortest u to 5. 

15: end if 

16: end for 

17: end for 

18: Find a pair (0, u) G 5. 

19: return u. 


Now we turn to the Algorithm that computes geodesics in a given 
subgroup. The algorithm keeps a set 5 of pairs {v, u) where n G P and 
u G F{hi, ..., hm)- We denote the initial set 5 = {(0, e)} by Sq and the set 
5 obtained after jth iteration by Sj. 

Lemma 6.4. Let w = .. .xP G F. Then for every j > 1 and every 

{v,u) G Sj: 

(PI) The graph Fq contains a path p from 0 to v such that fr{p) = 
xP ■ ■ ■ x^f,, v{p) is reduced, and v{p) = u. 

(P2) The word u is a shortest possible u = n{q) among all paths q in Tq 
from 0 to V with pi{q) = xP • • • xf. . 

(P3) Furthermore, ifT^ contains a path p from 0 to v such that n{P) = 
xP ... xP, then Sj contains a pair (v, u) for some u G F{hi ,..., hm)- 

Proof. Initially, 5 = {(0,e)}. To construct Si the algorithm constructs the 
set S'. It easy to see that 


S' = {(i(e), p(e)) I e G F s.t. o(e) = 0, fi{e) = x^)} 


and Si 

Si¬ 


s'. It follows from Lemma 6.3 that (PI), (P2), and (P3) hold for 
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Assume that Sj-i satisfies (PI), (P2), and (P3). To construct Sj the 
algorithm constructs a set 

s' = {(t(e), uv{e)) I (u, u) G Sj-i, e G P s.t. o(e) = u, /i(e) = )}. 

By induction assumption for every (u, u) G Sj-i there is a path pi i n Pq such 
that o{pi) = 0, t{pi) = V, p{pi) = By Lemma 6.3 for every 

e G P there is a pathp 2 in Pq satisfying o(e) = 0 (^ 2 ), ^(e) = t(p 2 ), p(e) = x^^. 
Hence o{pip 2 ) = 0, t{pip 2 ) = t(e), p{piP 2 ) = x^^ ■ ■ ■ x^^. Redncing pip 2 if 
necessary we obtain a path in Pq that delivers (PI) for (v,u) G S'. Since 
Sj C S', the property (PI) holds for every pair in Sj. 

Now we show (P2) and (P3). For a vertex u of Pq and for a given 1 < 


j < n, let p be a path in Pq from 0 to u with p(p) = x. 


v{p) 


reduced and shortest possible. Fixing a cancellation scheme for p{p) we 
split the path p into pi... Pj so that p{pa) = “ for every a = 1,..., j. Let 

v' = t{pi ■ ■ -pj-i). Since u{p) is rednced, it follows {v', ^{pi ■ ■ -pj-i)) G Sj-i. 


By Lemma 6.3, F contains an edge e from v' to v such that p(e) = . Hence 


S' contains a pair {v,v{p)). Since v{p) is shortest possible for the vertex v 
and the given j, it follows that |u| = |p(p)|. Therefore, the properties (P2), 
(P3) hold for Sj. □ 

Theorem 6.5. Let u = Geodesic(/ii,... ,hm,w). Then 7 ( 11 ) = w and u is 
a shortest with such property, i.e., u is geodesic for w. 

Proof. Let w = x^L ... xf". Assnme that w = h^f ■ ■ ■ and k is the least 

^1 Jl Jk 

possible. The graph F contains a circuit p at 0 such that p{p) = ■ ■ ■ h^j^, 

where every 1 < a < fe, is traced along the corresponding loop. By 
Lemma 6.4 the set Sn contains a pair (0,u) with |m| = k. Therefore the 
output is indeed a geodesic word for tc. □ 


Theorem 6.6. Algorithms'^ and\^ terminate in polynomial time. 

Proof. Algorithm starts with the graph Fq. To represent values on the 
edges it constructs a grammar consisting of terminal symbols hf,..., h^ 
and non-terminal symbols {Se | e G P'}. Adding an edge e corresponding 
to a bypass 61,62 adds a new non-terminal Se with production SeiSe 2 - It 
follows from m, it requires polynomial time to compute \iy{ei)i'{e2)\ for 
any pair of edges 61 , 62 . 

Algorithm processes n letters of the word w. It is easy to see that on 
every iteration |5| < |H| and |5'| < \ V\‘^. Choosing pairs {v,u) with the least 
|n| requires computing |tt|, which can be done in polynomial time. Hence, 
overall Algorithm has polynomial time complexity. □ 
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